Nice!

I looked at existing unit test frameworks in the past, but nothing seemed appropriate for us. There are not that many for C, and they were either overkill or too simple (just handful of ifdefs) so they didn't add any functionality. I thought writing our own is too annoying (or I was just lazy). But the framework is ~300 lines, that seems fine to me.

see also #1211

• The available "targets" seem to be a bit arbitary. Maybe we can try to put them into groups (perhaps similar to the grouping in #1211)?

  ./tests -print_tests
  Available tests (58):
      --------------------------------------------------
      [  1] selftest_tests
      [  2] all_proper_context_tests
      [  3] all_static_context_tests
      [  4] deprecated_context_flags_test
      [  5] scratch_tests
      [  6] int128_tests
      [  7] ctz_tests
      [  8] modinv_tests
      [  9] inverse_tests
      [ 10] hsort_tests
      [ 11] sha256_known_output_tests
      [ 12] sha256_counter_tests
      [ 13] hmac_sha256_tests
      [ 14] rfc6979_hmac_sha256_tests
      [ 15] tagged_sha256_tests
      [ 16] scalar_tests
      [ 17] field_half
      [ 18] field_misc
      [ 19] field_convert
      [ 20] field_be32_overflow
      [ 21] fe_mul
      [ 22] sqr
      [ 23] sqrt
      [ 24] ge
      [ 25] gej
      [ 26] group_decompress
      [ 27] ecmult_pre_g
      [ 28] wnaf
      [ 29] point_times_order
      [ 30] ecmult_near_split_bound
      [ 31] ecmult_chain
      [ 32] ecmult_constants
      [ 33] ecmult_gen_blind
      [ 34] ecmult_const_tests
      [ 35] ecmult_multi_tests
      [ 36] ec_combine
      [ 37] endomorphism_tests
      [ 38] ec_pubkey_parse_test
      [ 39] eckey_edge_case_test
      [ 40] eckey_negate_test
      [ 41] ecdh_tests
      [ 42] ec_illegal_argument_tests
      [ 43] pubkey_comparison
      [ 44] pubkey_sort
      [ 45] random_pubkeys
      [ 46] ecdsa_der_parse
      [ 47] ecdsa_sign_verify
      [ 48] ecdsa_end_to_end
      [ 49] ecdsa_edge_cases
      [ 50] ecdsa_wycheproof
      [ 51] extrakeys_tests
      [ 52] schnorrsig_tests
      [ 53] musig_tests
      [ 54] ellswift_tests
      [ 55] secp256k1_memczero_test
      [ 56] secp256k1_is_zero_array_test
      [ 57] secp256k1_byteorder_tests
      [ 58] cmov_tests
  

Thanks for the review jonasnick!

Thanks @furszy. I played around with this a little bit. It reduces the execution time on my machine from 26 seconds to 10 seconds (-jobs=16). Very nice!

Awesome :). I think we can actually do even better, will do some changes.

• It would be helpful if a helptext would be output when tests is run with -h or --help.

The help message was actually already there, but for -help only (with a single - and I forgot to add it to the PR description).
I just pushed support for -h as well.

• I think showing all the tests that have passed is a bit overkill. I'm already assuming that the tests pass if they do not show up in the output. I only need to see tests that don't pass.

Sure. Will hide the logging behind a -log option (or -silent if we want the opposite behavior).
From my experience, logging sometimes helps spot regressions or areas that can be improved (like when a test suddenly takes longer than usual), and it’s also reassuring to see them run and pass when you have a large number of them.
I’ve also been thinking about adding per-test execution time loggings, might be a good opportunity to include it too.

• Maybe a future PR can autodetect the number of cores and set -jobs automatically by default?

I'm not sure we want that. Sequential execution is usually "standard" on any system because we don’t know what else the user might be running. Picking a number of parallel tasks automatically (even if it is a low number) could hang the CPU or even make it run slower than sequential if the system is overloaded.

• There's an -iter command line flag, but the test output shows "test count". It would be better if we were consistent.

Sure 👍🏼. That was carried over from the previous code; will improve it.

The available "targets" seem to be a bit arbitary. Maybe we can try to put them into groups (perhaps similar to the grouping in #1211)?

Yeah. Just reworked the framework to support registering and running groups of tests in a generic manner. This means we can now run specific tests and/or specific groups of tests via the -target/-t arg.

On top of that, made the framework reusable across binaries and improved the overall API (we can now easily connect the tests_exhaustive binary to it too — probably something for a follow-up), along with improvements to the consumers’ structure, enforcing consistency and a specific pattern that all consumers will follow.

Other than that, the -print_tests option was improved to display the available modules and tests.

A simple usage example:
./tests -print_tests → pick any module (like field) and run: ./tests -t=field
You can also combine this with -j=<num_workers>, plus specify multiple targets as needed.

To fix Autotools:

diff --git a/Makefile.am b/Makefile.am
index d379c3f..dc79857 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -123,7 +123,7 @@ if USE_TESTS
 TESTS += noverify_tests
 noinst_PROGRAMS += noverify_tests
 noverify_tests_SOURCES = src/tests.c
-noverify_tests_CPPFLAGS = $(SECP_CONFIG_DEFINES) -DSUPPORTS_CONCURRENCY=$(SUPPORTS_CONCURRENCY)
+noverify_tests_CPPFLAGS = $(SECP_CONFIG_DEFINES) $(TEST_DEFINES)
 noverify_tests_LDADD = $(COMMON_LIB) $(PRECOMPUTED_LIB)
 noverify_tests_LDFLAGS = -static
 if !ENABLE_COVERAGE
diff --git a/configure.ac b/configure.ac
index 55eebdf..6028ee2 100644
--- a/configure.ac
+++ b/configure.ac
@@ -447,8 +447,8 @@ fi
 if test "x$enable_tests" != x"no"; then
   AC_CHECK_HEADERS([sys/types.h sys/wait.h unistd.h])
   AS_IF([test "x$ac_cv_header_sys_types_h" = xyes && test "x$ac_cv_header_sys_wait_h" = xyes &&
-         test "x$ac_cv_header_unistd_h" = xyes], [SUPPORTS_CONCURRENCY=1])
-  AC_SUBST(SUPPORTS_CONCURRENCY)
+         test "x$ac_cv_header_unistd_h" = xyes], [TEST_DEFINES="-DSUPPORTS_CONCURRENCY=1"], TEST_DEFINES="")
+  AC_SUBST(TEST_DEFINES)
 fi
 
 ###

Thanks @hebasto! Updated with autotools patch.

• A big feature I'm missing is integration into the build framework, such that e.g. test invocations get automatically split by module, and have ctest --test_dir=build -j16 run the tests binary once for each module. This would automatically give well-balanced parallellism, that's automatically integrated into CI, even for projects that include it (Bitcoin Core). Is something like that planned?

See #1734 (comment).

Updated per feedback. Thanks!

1. Removed testrand_finish() call.
2. Added a more descriptive error message for the 255 tests current cap.